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(54) Abstract Title 

Efficient capture of computer screens 

(57) A method for capture of computer screens in a 
sequence of frames. A first set of windows appearing in a 
first frame in the sequence is identified, each window 
having respective first-frame window characteristics and 
contents. A description of the first set of windows is 
encoded, indicative of the appearance of the computer 
screen in the first frame. In a second frame in the 
sequence, a second set of windows is identified, having 
respective second-frame window characteristics and 
contents, the second set including one or more windows 
corresponding respectively to one or more of the first set 
windows. Transformations are determined and applied to 
first-frame window characteristics of the windows in the 
first set to generate second-frame window characteristics 
of corresponding second set windows. A description of 
the second set of windows is encoded, including the 
determined transformations, for use in reconstructing the 
computer screen as it appeared in the second frame. 
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EFFICIENT CAPTURE OF COMPUTER SCREENS 

The present invention relates generally to computer software, and 
specifically to programs enabling capture, storage and communication of the 
contents of computer displays. 

There are a variety of computer screen capture tools known in the 
art. These tools enable the contents and appearance of a computer screen 
to be captured, or recorded, more or less in real-time. Generally, a 
sequence of screens is captured and is then stored to disk and/or 
transferred to another computer. Screen capture tools of this sort are 
useful, for example, in educational applications and in training and 
promotional demonstrations. Screen capture is also used by computer remote 
control tools. 

Screen capture products for education, training and promotion include 
Lotus ScreenCam (http://www.lotus.com/ screencam) , Hyperionics HyperCam 
(http: //www.hyperionics. com) and TechSmith SnagIt/32 

(http://www.techsmith.com). These products enable a user to record the 
contents of a computer screen to a file, while the computer is carrying out 
another program, and then to reproduce the recorded screen content from the 
file. They evidently work by encoding a bitmap image of the entire 
contents of the screen. Multiple screens in sequence may be recorded by 
encoding the differences between successive screens. This approach usually 
generates large amounts of processed data and very large output files. As 
a result, users may be limited to working at very slow refresh rates, on 
the order of one or a few frames per second, if they wish to record a full, 
active computer screen. Transferring the output files over a low-bandwidth 
computer network may be even slower. The alternative is to compromise on 
the content of the recording, typically by reducing the color resolution, 
by recording only a portion of the screen, or by simplifying the screen 
contents, by reducing the number of windows that are open on screen, for 
example . 

OPTX International ScreenWatch (http: //www. screenwatch.com) uses an 
alternative approach of capturing data sent to the computer' s display 
driver, in this case a proprietary driver developed for this purpose by 
OPTX. The display driver runs on a Microsoft Windows NT computer, which 
conveys the data to a separate client computer for recording. The data are 
stored in a proprietary format, which can subsequently be played back using 
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a dedicated player program. The alternative approach, employed by 
ScreenWatch enables faster, more efficient screen capture, but is limited 
to the complex, proprietary operating environment for which it was 
designed . 

5 

Computer remote control tools include Symantec PCAnywhere 
(http://www.symantec.com/region/can/eng/ product /pc anywhere) and 
LapLink.com LapLink (http://www.travsoft.com/products/llpro). These 
products enable a remote user to control a host computer and observe the 
10 screen contents of the host. They are not capable of keeping up with large 

or rapid changes on the host computer screen, even at high transmission bit 
rates between the host and remote computers. 

The present invention provides, in a first aspect, a method for 
15 capture of computer screens in a sequence of frames, including: identifying 

a first set of one or more windows appearing in a first frame in the 
sequence, each window in the set having respective first -frame window 
characteristics and window contents; encoding a description of the first 
set of windows indicative of the appearance of the computer screen in the 
20 first frame; identifying in a second frame in the sequence a second set of 

one or more windows having respective second- frame window characteristics 
and window contents, the second set including one or more windows 
corresponding respectively to one or more of the wind.ows in the first set; 
determining one or more transformations applied to the first -frame window 
25 characteristics of the windows in the first set to generate the 

second- frame window characteristics of the corresponding windows in the 
second set; and encoding a description of the second set of windows 
including the determined transformations, for use in reconstructing the 
computer screen as it appeared in the second frame. 

30 

Preferably, identifying the first set of windows includes identifying 
windows generated in accordance with an operating system of the computer, 
which associates each window with a respective function of the computer, 
such that the contents of the windows are determined by the respective 

35 functions. Most preferably, the respective functions include applications 

running under the operating system. Further preferably, determining the 
transformations includes defining transformations applied by the operating 
system, and which are applicable to different windows associated with 
different functions, generally irrespectively of the functions, 

40 Preferably, identifying the second set of windows includes querying the 

operating system regarding the characteristics of the windows. 
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Additionally or alternatively, identifying the second set of windows 
includes intercepting events generated by the operating system. 

In a preferred embodiment, identifying the first set of windows 
5 includes processing an image of the screen to identify the windows. 

Preferably, determining the transformations includes defining a set 
of typical transformations, which are applicable to alter the window 
characteristics of the one or more windows, generally irrespectively of the 

10 contents of the windows. In preferred embodiments, the typical 

transformations are selected from a group of transformations including 
moving and resizing a window; minimizing, restoring and maximizing the size 
of a window; changing a Z-order of the windows, according to which two or 
more of the windows are overlaid one upon another on the screen; and 

15 scrolling and panning the window contents. 

Preferably, encoding the description of the first set of windows 
includes encoding the first- frame window characteristics and the respective 
contents of the windows in the first set, and encoding the description of 

2 0 the second set of windows includes encoding the determined transformations 

and encoding changes in the contents of the windows in the second set with 
respect to the contents of the corresponding windows in the first set. In 
a preferred embodiment, encoding the changes in the contents of the windows 
includes applying different encoding schemes to the contents of different 

25 ones of the windows, wherein applying the different encoding schemes 

includes applying a video compression scheme to the contents of at least 
one of the windows, and embedding resultant compressed video data in the 
encoded description of the windows. Additionally or alternatively, 
applying the different encoding schemes includes applying different levels 

30 of encoding resolution to different ones of the windows in the second set. 

Preferably, identifying the first and second sets of windows includes 
identifying windows associated with respective functions of the computer, 
wherein the identified windows include one or more user interface windows 
35 generated inside other identified windows for the purpose of controlling 

the functions associated therewith. 

In a preferred embodiment, the method includes identifying first and 
second sets of one or more icons in the first and second frames, 
40 respectively, and determining transformations applied to the icons in the 

first frame to generate the icons in the second frame, to be encoded along 



with the description of the second set of windows. Preferably, the first 
and second sets of icons include a cursor. 

In another preferred embodiment, encoding the first and second 
descriptions includes transferring the encoded descriptions over a 
communication link to a recipient computer. Most preferably, encoding the 
descriptions includes encoding the descriptions in a plat form- independent 
format . 

Alternatively or additionally, encoding the first and second 
descriptions includes storing the encoded descriptions in a memory. 

There is also provided, in a second aspect of the present invention, 
a method for reconstructing captured computer screens, including: 
receiving an encoded description of a first set of one or more windows, 
having first -frame characteristics and window contents, which appeared on 
the computer screen in a first captured frame; receiving an encoded 
description of a second set of one or more windows, having second- frame 
characteristics and window contents, which appeared on the computer screen 
in a second captured frame, subsequent to the first frame, the description 
of the second set of windows including a description of one or more 
transformations applied to the first-frame characteristics of at least one 
of the windows in the first set to derive the second- frame characteristics 
of a corresponding window in the second set; and reconstructing the second 
captured frame responsive to the encoded descriptions of the first and 
second sets of windows. 

Preferably, reconstructing the second captured frame includes 
decoding the encoded description of the first set of windows to determine 
the first- frame characteristics thereof, and applying the one or more 
transf oirmations described in the description of the second set of windows 
to transform the first-frame characteristics into the second-frame 
characteristics of the at least one corresponding window. Most preferably, 
the encoded description of the second set of windows further includes 
encoded changes in the contents of the windows in the second set with 
respect to the contents of the at least one corresponding window in the 
first set, and reconstructing the second captured frame includes 
reconstructing the contents of the windows in the second set responsive to 
the encoded changes. In a preferred embodiment, the encoded description of 
the second set of windows includes compressed video data in a standard 
media format, and reconstructing the second captured frame includes 



invoking a standard media player to reconstruct video images in one of the 
windows * 

Preferably, reconstructing the second captured frame includes 
reconstructing the first and second sets of windows substantially 
independently of an operating system under which the windows were 
generated, wherein reconstructing the first and second sets of windows 
includes operating a platform- independent screen player. 

In a third aspect, the present invention provides apparatus for 
capture of computer screens in a sequence of frames, including: a display; 
and a processor, which is adapted to identify a first set of one or more 
windows appearing on the display in a first frame in the sequence, each 
window in the set having respective first-frame window characteristics and 
window contents, and to encode a description of the first set of windows, 
indicative of the appearance of the computer screen in the first frame, and 
to identify, in a second frame in the sequence a second set of one or more 
windows appearing on the display, having respective second- frame window 
characteristics and window contents, the second set including one or more 
windows corresponding respectively to one or more of the windows in the 
first set, and to determine one or more transformations applied to the 
first-frame window characteristics of the windows in the first set to 
generate the second- frame window characteristics of the corresponding 
windows in the second set, and to encode a description of the second set of 
windows including the determined transformations, for use in reconstructing 
the computer screen as it appeared in the second frame. 

In a preferred embodiment, the processor is adapted to be coupled via 
a communication link to transfer the encoded descriptions to a recipient 
computer, which reconstiructs the second frame responsive to the encoded 
descriptions of the first and second sets of windows. 

In another preferred embodiment, the processor includes a memory 
adapted to store the encoded first and second descriptions. 

In a fourth aspect, the present invention provides apparatus for 
reconstructing captured computer screens, including: processor, which is 
adapted to receive an encoded description of a first set of one or more 
windows, having first -frame characteristics and window contents, which 
appeared on the computer screen in a first captured frame, and to receive 
an encoded description of a second set of one or more windows, having 



second-frame characteristics and window contents, which appeared on the 
computer screen in a second captured frame, subsequent to the first frame, 
the description of the second set of windows including a description of one 
or more transformations applied to the first- frame characteristics of at 
least one of the windows in the first set to derive the second- frame 
characteristics of a corresponding window in the second set, and to 
reconstruct the first and second captured frames responsive to the encoded 
descriptions of the first and second sets of windows; and a display, which 
is adapted to be driven by the processor to display the reconstructed first 
and second frames. 

The present invention, in a fifth aspect, provides a computer program 
comprising computer program instructions to, when loaded into a computer 
system and executed, cause the computer to perform the steps of a method 
according to the first aspect. Preferably, the program instructions are 
provided in a platform- independent format, A preferred embodiment of the 
present invention is as a computer software product for capture of computer 
screens in a sequence of frames, the product including computer- readable 
media in which program instructions are stored, which instructions, when 
read by a computer, cause the computer: to identify a first set of one or 
more windows appearing in a first frame in the sequence, each window in the 
set having respective first -frame window characteristics and window 
contents and to encode a description of the first set of windows, 
indicative of the appearance of the computer screen in the first frame, and 
to identify in a second frame in the sequence a second set of one or more 
windows having respective second- frame window characteristics and window 
contents, the second set including one or more windows corresponding 
respectively to one or more of the windows in the first set and to 
determine one or more transformations applied to the first -frame window 
characteristics of the windows in the first set to generate the 
second-frame window characteristics of the corresponding windows in the 
second set, and to encode a description of the second set of windows 
including the determined transformations, for use in reconstructing the 
computer screen as it appeared in the second frame. 

In a sixth aspect, the present invention provides a computer program 
comprising computer program instructions to, when loaded into a computer 
system and executed, cause the computer to perform the steps of a method 
according to the second aspect. Preferably, the program instructions are 
provided in a platform- independent format. 



It is preferrred that a computer program of the sixth aspect be 
embodied in a computer software product for reconstructing captured 
computer screens, the product including computer- readable media to be read 
by a computer that receives an encoded description of a first set of one or 
more windows belonging to a first captured frame, the windows having 
first-frame characteristics and window contents, and an encoded description 
of a second set of one or more windows belonging to a second captured 
frame, subsequent to the first frame, the windows having second- frame 
characteristics and window contents, the description of the second set of 
windows including a description of one or more transformations applied to 
the first -frame characteristics of at least one of the windows in the first 
set to derive the second- frame characteristics of a corresponding window in 
the second set, wherein program instructions are stored in the 
computer- readable media, which instructions, when read by the computer, 
cause the computer to reconstruct the second captured frame responsive to 
the encoded descriptions of the first and second sets of windows. 

Embodiments of the present invention may reduce the volume of data 
required to capture complex computer screen contents. 

Embodiments of the present invention may increase the frame rate at 
which computer screen contents can be captured. 

Embodiments of the present invention provide tools for computer 
screen capture that are applicable to a wide range of platforms and can be 
played back by a platform- independent player. 

In preferred embodiments of the present invention, a computer screen 
capture tool treats windows on the computer display as objects, and records 
changes in the characteristics of these objects and relations among them 
from frame to frame. Preferably, a group of typical transformations of the 
windows is defined, and these typical transformations are encoded and 
recorded separately from other changes in the window contents. Generally 
speaking, the typical transformations are defined by an operating system of 
the computer, and they are therefore common to windows running different 
applications and can be encoded very compactly for all of the windows on 
the screen. In this way, the amount of bitmap data that must be recorded 
is substantially reduced relative to screen capture tools known in the art. 

Displays with multiple active windows can thus be recorded and 
transmitted in real time, with high temporal resolution (i.e., high frame 
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refresh rates) , as well as full color definition and detail, while the 
computer is carrying out application tasks. No special resources are 

needed, in contrast to products known in the art such as the 

above-mentioned ScreenWatch. Different encoding schemes can be applied to 
the contents of different windows, depending on the type of contents (for 
example, video as opposed to text) . In some preferred embodiments, 
movements of other on-screen objects, such as a mouse-driven cursor and 
other icons, are also encoded using typical transformations. 

To play back the recorded screens, the windows and other objects are 
reconstructed in each successive frame by applying the encoded typical 
transformations to the windows and objects in the preceding frame. The 
window contents are then reconstructed inside the windows. Preferably, a 
screen player program for reconstructing the recorded screens is 
independent of the operating system of the recording computer. Most 
preferably, the screen player is provided in a platform- independent form, 
for example, in the Java language. 

Preferred embodiments of the present invention are useful in a range 
of applications, including demonstrations and presentations, education and 
remote control, as described in the background of the invention above. 

A preferred embodiment of the present invention will be now be 
described by way of example only, with reference to the accompanying 
drawings in which: 

Fig. 1 is a schematic, pictorial illustration showing apparatus for 
capturing and replaying computer screens, in accordance with a preferred 
embodiment of the present invention. 

Fig. 2 is a schematic illustration of a computer screen captured by 
the apparatus of Fig. 1, in accordance with a preferred embodiment of the 
present invention; 

Fig. 3 is a flow chart, which schematically illustrates a method for 
capturing computer screens, in accordance with a preferred embodiment of 
the present invention; 

Fig. 4 is a flow chart, which schematically illustrates a method for 
replaying computer screens, in accordance with a preferred embodiment of 
the present invention; and 



Fig. 5 is a flow chart, which schematically illustrates details of a 
method for screen image reconstruction, in accordance with a preferred 
embodiment of the present invention. 

Fig, 1 is a schematic, pictorial illustration showing screen capture 
apparatus 20 for capturing computer screens and playback apparatus 40 for 
reconstructing and playing back the captured screens, in accordance with a 
preferred embodiment of the present invention. Preferably, both apparatus 
2 0 and apparatus 4 0 comprise computers, typically personal computers, each 
comprising a processor 22, a keyboard 24, a pointing device, such as a 
mouse 26, and a display 28. Capture apparatus 20 runs operating system 
software, such as Microsoft Windows or other similar operating systems 
known in the art, which generates a plurality of windows 32, 34 36 on 
display 28. Typically, each window displays data and allows user 
interaction with a different, respective software application running on 
apparatus 20, or with different instances of a given application. 

As described in detail hereinbelow, capture apparatus 20 runs a 
screen capture program, which encodes the images shown on display 2 8 for 
subsequent playback. Preferably, encoded data corresponding to the display 
images are conveyed over a commimication link 38, such as a computer 
network, for playback on apparatus 40. Alternatively or additionally, the 
encoded data are recorded in a memory of apparatus 20, typically on a hard 
disk 30, The data recorded on disk 3 0 may also be played back on the same 
apparatus 2 0 on which the screens were captured. The programs required by 
processors 22 for capturing and reconstructing the screen images may be 
downloaded to apparatus 20 and/or 4 0 in electronic form via a network, for 
example, or they may alternatively be supplied on tangible media, such as 
CD-ROM. 

Fig. 2 is a schematic illustration showing details of display 28 as 
captured by apparatus 20, in accordance with a preferred embodiment of the 
present invention. The display shows a family 48 of windows, including 
open windows 32, 34 and 36 and another window 52 which is minimized and 
displayed only as an icon, although the application associated with the 
window may continue to run. Transformations such as minimization, 
maximization, restoration and closing of each of the windows are typically 
effected using controls 54, as are known in the art. Each of the windows 
can also be moved and resized, generally by using mouse 26 to manipulate a 
cursor 56 on screen. At any point that a window is open on display 28, its 
size and position are defined by its corners 50, wherein assuming the 
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window to be rectangular, the coordinates of two of the corners are 
sufficient to fully define the size and position. 

The windows in family 48 are also characterized by a Z-order, which 
5 determines their respective priorities when two or more windows overlap. 

In the case shown in the figure, the order is window 32, followed by window 
34, followed by window 36, although of course, the order commonly changes 
from time to time. 

10 As mentioned above, each of the windows in family 48 is typically 

(although not necessarily) associated with a different application. By way 
of example, window 32 is running a graphic application, window 34 is 
displaying a real-time video image, and window 3 6 is running a text 
application. Window 3 6 includes a scroll bar 58, which enables a user to 

15 scroll through the document shown in the window. The contents of each of 

the windows are updated regularly by the applications associated therewith. 
The applications may also include other effects, particularly sound, which 
is typically played in conjunction with the display in the respective 
window . 

20 

Separate and apart from application-specific changes in the window 
contents, there are common transformations that can be applied to any of 
the windows or at least to a range of different applications. Such 
transformations are generally implemented in the operating system, although 
2 5 some of them may be generated by application or utility programs. A list 

of such transformations, referred to herein as typical transformations, is 
presented by way of example, but not limitation, in Table I below. 

TcLble I - typical transformations 



30 


• 


Move window 




• 


Resize window 




• 


Minimize (iconize) window 




• 


Maximize window 




• 


Restore window 


35 


• 


Change Z -order of windows 




• 


Scroll window contents 




• 


Pan window contents 




• 


Change colour palette 
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other transformations may also be classified as typical, for example, 
inversion of tiie contents of a window. These and other transformations can 
also be applied to non- rectangular windows or overlays, although the sizes 
and positions of such windows may need to be represented by more than just 
the corner positions used for standard rectangular windows. Movements of 
cursor 56 and other on-screen icons can likewise be classified as '"move" 
operations, similar to moving of windows. 

Fig. 3 is a flow chart that schematically illustrates a method for 
capturing and encoding computer screens using typical transformations, in 
accordance with a preferred embodiment of the present invention. The 
method is described with reference to window family 48, shown in Fig. 2, on 
a personal computer running a Microsoft Windows operating system, but it 
will be understood that the principles of this method are equally 
applicable to other types of windows and other operating systems and 
applications . 

For each screen to be captured, at each capture time, or frame time 
ti, apparatus 2 0 identifies the windows and other objects shown on display 
28, at a find window step 60. In the example of Fig. 2, these windows and 
objects would include windows 32, 34, 36 and 52 (which is ^^iconized" ) / as 
well as cursor 56. Optionally, other icons and window- like objects are 
also captured, for example, menu windows and sub-windows that are opened 
within the client areas of the application windows. The characteristics of 
the windows and objects, including their location, size and Z-order, are 
recorded, so as to define window family 48 at time ti, referred to herein as 
FW(ti) , 

Formally, FW(ti) preferably contains the group of windows Wi, W2, 
WN(i>, each window, dependent on the time instance ti characterized by the 
following parameters: 

• A set of corners 50 . 

• A bit value b(0,l) indicating whether the window is iconized in the 
current f rame . 

• A Z-order position. In this regard, FW{ti) may be regarded as a 
directed graph (digraph) , wherein there is a vertex in the graph 
corresponding to each window W3, and directed edges of the graph 
connecting the vertices, dependent on the Z-order relation between the 
respective windows . 
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• The window content. Typically tiie content is represented as a bitmap, 
but it may also be captured and stored in other, application -specific 
formats, as described further hereinbelow. 



5 Preferably, FW(ti) is constructed by querying the operating system 

and, optionally, the application software running on processor 22 of 
apparatus 20 as to the window parameters. In a preferred embodiment of the 
present invention, the queries are made using application program interface 
(API) commands available for the Windows operating system, including 

10 EnumWindows, GetWindowRect , GetDeviceCaps , GetWindowDC, ReleaseDC, 

Islconic, GetTopWindow, and IsWindowVisible , Alternatively, other methods 
may be used to identify the windows and extract the required parameters* 
For example, a window procedure subclassing technique may be used to 
intercept the messages posted or sent to the windows, as described in the 

15 WIN32 Programmer's Reference (Microsoft Press, 1993), which is incorporated 

herein by reference. Alternatively, a pixel image of display 2 8 may be 
processed, using image processing methods known in the art, in order to 
identify rectangular shapes corresponding to the windows on screen. 

20 It should be understood that while the description herein of the 

method illustrated in Fig. 3 makes reference to construction of the family 
of windows FW(tJ in each frame, it is generally not necessary to construct 
FW(ti) aJb Initio except in the initial frame at to. Rather, at each time ti 
(except to), resources needed for constructing FW(ti) are obtained from the 

25 preceding window family FW(ti.i). 

Each frame in the sequence of screen images to be captured (except 
for the first frame, of course, at time to) , is compared to the preceding 
frame, in a compare step 62. This step classifies the windows in family 
3 0 4 8, FW{ti), into three groups: 

1. Windows that were also present in the preceding frame FW(ti.i) 
2 . Windows that were in the preceding frame but are absent from the 
current frame. 

35 3 . Windows that appear in the current frame, but were absent in the 

preceding frame. 



40 



In an eliminated windows encoding step 63, information regarding the 
windows in group 2 (such as the indices of the graph vertices corresponding 
to these windows in FW(ti_i)) is encoded. By eliminating the windows in 
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group 2 from the set of windows in FW(ti_i) (group 1), an intermediate 
family FWl is defined containing the windows that appear in both the 
current frame and the preceding frame, with their parameters at time ti_i. 
Changes to the windows of FWl can be characterized by typical 
transfoinmations, as described further hereinbelow. Treatment of the 
windows in group 3 is described further hereinbelow. 

The windows in the intermediate family FWl and their parameters are 
compared to their counterparts in FW(ti) at a typical transformation 
encoding step 64 and a residual transformation encoding step 66 . In step 
64, those changes in the windows that are capable of definition as typical 
transformations, such as those listed in Table I, are identified and 
encoded. For example, in a successive frame to that shown on display 28 in 
Fig. 2, window 32 might be shifted, window 34 might be closed or iconized, 
and window 36 might be scrolled. In this case, the shift can be encoded 
symbolically as SHIFT (A, X, Y) , wherein A identifies the window, and X and Y 
are the displacement coordinates in pixels. Closing or iconizing of window 
34 can be encoded respectively as CL.OSE(B) or ICONIZE (B, ICON, X, Y) , wherein 
ICON refers to the minimized representation of the window on screen, and 
(X,Y) is its position. Scrolling of window 36 can be encoded as 
SCROLL (C, Y, BMP) , wherein Y is the scrolling displacement (which may be 
positive or negative) , and BMP points to a bitmap of height Y representing 
the content added to the top or bottom of the window at time ti . 

It will be understood that these are merely representative examples, 
and other possible types of transf oirmations and schemes for representing 
such transformations will be apparent to those skilled in the art. What is 
important to note is the tremendous savings in data volume req[uired to 
encode the contents of display 28 afforded by the present invention, by 
comparison with indiscriminate bitmap screen capture. In a bitmap 
representation of the entire display, a shift of window 32, for example, 
will require that substantially all of the pixels corresponding to the 
window be rewritten, at both the previous and current positions of the 
window, typically generating tens to hundreds of thousands of data bytes. 
The present invention enables the shift to be recorded using only a few 
bytes of data. 

Preferably, the typical transformations recorded at step 64 also 
include transformations of cursor 56 and other on-screen icons. In the 
case of the cursor, the transformations include SHIFT and changes in the 
form on the corresponding icon (point left, point right, text cursor, etc.) 
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Application of the recorded typical transformations to the windows in 
the intermediate family FWl will result in the generation of a transformed 
intermediate family FW2 . At step 66, the residual transformations to the 
windows in FW2 , which could not be encoded as typical transformations and 
which must be carried out in order to transform these windows into the 
corresponding members of FW(ti) , are also encoded. Generally, although not 
necessarily, the preferred method for encoding a given window in FW{ti) is 
by encoding the changes in the content of the window relative to its 
counterpart in FW2 , which reflects the result of typical transformations 
applied to the window content. Various methods are known in the art for 
such encoding, and it is an advantage of the present invention that 
different encoding methods and parameters may be applied to the different 
windows . 

In one preferred embodiment of the present invention, the bitmaps of 
one or more of the windows in FW{ti) (or of all of the windows) are compared 
to their counterparts in FW2 , and changes in the pixels are recorded, pixel 
by pixel. The resultant difference bitmap may be compressed, using any 
suitable method known in the art, such as run length encoding or LZW 
encoding. This type of encoding is particularly suitable for windows whose 
contents change relatively slowly, such as graphic window 32 or tejxt window 
36 . 

Alternatively or additionally, when the contents of a window change 
rapidly, as will be the case for video window 34 , methods of video encoding 
are preferably applied, for example, MPEG and other compression algorithms 
known in the art. In a preferred embodiment, the MPEG or other video data 
are recorded separately from the contents of non-video windows. Most 
preferably, such video data are recorded in their original compressed data 
format and at the original frame rate of the video images that were 
generated by the application running in window 34, which may be different 
from the frame rate at which the other screen contents are captured. 

In another preferred embodiment of the present invention, different 
encoding priorities are assigned to different windows in family 48, 
depending on their Z-order or on the applications running in the windows, 
for example. Thus, it is possible to encode changes to the bitmap contents 
of window 32, which is the top window in Fig, 2, in every recorded frame, 
while changes to bottom window 3 6 are encoded only once every few frames. 
Different compression schemes may also be applied to different windows, 
with lossy compression applied to low-priority windows. In an extreme 
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case, such as a demanding motion video or graphic application, 
lower-priority frames may be frozen altogether. By the same token, the 
methods of the present invention may be applied in a straightforward manner 
to capture just a single window or a limited subset of the windows of 
5 interest, by recording only the contents and typical transformations 

applied to the window or windows of interest, while ignoring the remaining 
screen contents. Preferably, a user interface is provided on apparatus 20 
to enable the user to select different screen capture parameters to be 
applied to different ones of the windows. 

10 

In still another preferred embodiment of the present invention, data 
are captured representing the contents of a given window or windows without 
reference to succeeding frames. This representation is useful particularly 
in data streaming applications . 

15 

Returning now to the windows in group 3 , which were absent in 
FW{ti-i), these windows are preferably captured and encoded aJb Initio, at an 
encode new windows step 68. (At the first frame, all of the windows in 
FW(to) are in group 3.) Step 68 includes finding corners 50 of each new 
20 window, its bit value b, Z-order position and bitmap contents. The graph 

of the window family is updated to add these new windows. 

The encoded typical and residual transformations, along with the new 
window information, are conveyed to an output data stream, at an output 

25 stream step 70. To the extent that the window contents include video data 

in a compressed video format, such as that shown in window 34, the 
compressed video is embedded in the output stream, preferably interleaved 
with the other screen capture data. In this case, the representation of 
the corresponding window in the screen capture data includes a pointer to 

30 the interleaved video stream. Audio data associated with window 34 or with 

another active window on display 28 can be interleaved in similar fashion. 
The output data stream can be stored to disk 30 or transferred immediately 
over link 3 8 for playback on apparatus 40, Meanwhile, apparatus 2 0 returns 
to step 60 to capture and encode the next frame, 

35 

The output data stream is read by a compatible screen player running 
on apparatus 40, as described further hereinbelow. Preferably, the data 
are forTtiatted in a manner that is plat form- independent , so that it will be 
possible to replay the screens even if apparatus 4 0 is running a different 
40 operating system from apparatus 20. 
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Fig, 4 is a flow chart that schematically illustrates a method for 
reconstruction of screen images, in accordance with a preferred embodiment 
of the present invention. The data stream captured by apparatus 2 0 is 
received by apparatus 40 over link 38 at a receive input step 80. The 
stream may be conveyed to apparatus 4 0 for purposes of demonstration, 
training or education or, alternatively, apparatus 40 may be controlling 
the operation of processor 22 by remote control over link 38, as is known 
in the art, and receiving the screen images in this context. Further 
alternatively, the data stream may have been stored on disk 30 and later 
recalled from the disk by apparatus 20. 

The data are received by a compatible screen player, most preferably 
a platform- independent Java player. The player first reads the data in the 
stream relating to the initial frame at time to, and uses the data to 
reconstruct an initial window family FW(to)/ at an initial window 
reconstruction step 82, The player then reads and reconstmacts the 
application-specific content that is displayed in each of the windows, at 
an initial content reconstruction step 84 . To the extent that any of the 
windows; such as window 34, contain data encoded in a standard compressed 
media format, such as a video or audio format as described hereinabove, the 
screen player preferably invokes an appropriate standard media player, 
compatible with the compressed video or audio. For video data, the video 
player runs and displays the video in window 34 under the control of the 
screen player. 

Once the initial frame has been reconstructed, the screen player 
receives reconstruction information Di for each of the subsequent frames in 
succession at a receive information step 86. For each i = 1, 2, N, 
includes information regarding the windows that existed in the preceding 
frame (at time ti.i) but were eliminated in the current frame (at time ti) , 
along with the encoded typical transformations, the encoded residual 
transformations and the encoded new windows. At an eliminated windows 
decoding step 87, the identification of the eliminated windows is decoded. 
The typical transformations for each frame are decoded in a typical 
transformation decoding step 88. The residual transformations are 
similarly decoded, at a residual transformation decoding step 90. The 
decoded information is then used to reconstruct window family 48, in a 
screen reconstruction step 92 . 

Fig. 5 is a flow chart that schematically illustrates a method of 
screen reconstruction used at step 92 , in accordance with a preferred 
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embodiment of the present invention. For each i = 1, 2, N, the 
information Di is used to reconstruct the windows in FW(ti) at a window 
family reconstruction step 94. The information regarding the windows that 
were in the preceding frame but are absent from the frame that is currently 
5 being reconstructed is used to construct the interim window family FWl 

(consisting of the windows present at both times ti.i and ti with the window 
parameters at time ti-J , which is included in the preceding family FW{ti.a), 
The decoded typical transformations are applied to FWl to generate the 
interim window family FW2 , from which the new window family FW{ti) is 
10 derived. Following this step, the windows in FW(ti) are arranged in their 

proper position and relations for frame i, with the exception of any new 
window that may have been added in this frame. 

The stored residual transformations, defining the contents of the 
15 windows in frame i relative to their content in frame i-1, are now applied 

to reconstruct the window contents. Finally, using the decoded information 
regarding any new windows in this frame, reconstruction of the window 
family FW(ti) is completed, preferably including the entire screen contents 
at time ti . 

20 

The reconstructed windows are passed to a reconstruct screen content 
step 96, at which the reconstructed windows are assembled into a complete 
screen picture. Alternatively, steps 94 and 96 could proceed in parallel. 
As noted above, compressed video data are written into their appropriate 
25 window, as well. These steps are repeated in succession for each frame 

until the entire captured frame sequence has been played back. 
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CLAIMS 

1. A method for capture of computer screens in a sequence of frames, 
comprising : 

identifying a first set of one or more windows appearing in a first 
frame in the sequence, each window in the set having respective first-frame 
window characteristics and window contents; 

encoding a description of the first set of windows indicative of the 
appearance of the computer screen in the first frames- 
identifying in a second frame in the sequence a second set of one or 
more windows having respective second- frame window characteristics and 
window contents, the second set including one or more windows corresponding 
respectively to one or more of the windows in the first set; 

determining one or more transformations applied to the first-frame 
window characteristics of the windows in the first set to generate the 
second-frame window characteristics of the corresponding windows in the 
second set ; and 

encoding a description of the second set of windows including the 
determined transf onnations , for use in reconstructing the computer screen 
as it appeared in the second frame . 

2. A method as claimed in claim 1, wherein identifying the first set of 
windows comprises identifying windows generated in accordance with an 
operating system of the computer, which associates each window with a 
respective function of the computer, such that the contents of the windows 
are determined by the respective functions , 

3. A method as claimed in claim 2, wherein the respective functions 
comprise applications running under the operating system. 

4. A method as claimed in claim 2 or claim 3, wherein determining the 
transformations comprises defining transformations applied by the operating 
system, and which are applicable to different windows associated with 
different functions, generally irrespectively of the functions. 



5. A method as claimed in any of claims 2 to 4, wherein identifying the 
second set of windows comprises querying the operating system regarding the 
characteristics of the windows. 

6. A method as claimed in any of claims 2 to 5, wherein identifying the 
second set of windows comprises intercepting events generated by the 
operating system. 

7. A method as claimed in any preceding claim, wherein identifying the 
first set of windows comprises processing an image of the screen to 
identify the windows. 

8 . A method as claimed in any preceding claim, wherein determining the 
transformations comprises defining a set of typical transformations, which 
are applicable to alter the window characteristics of the one or more 
windows, generally irrespectively of the contents of the windows. 

9. A method as claimed in claim 8, wherein the typical transf onnations 
are selected from a group of transformations consisting of moving and 
resizing a window. 

10. A method as claimed in claim 8 or claim 9, wherein the typical 
transformations are selected from a group of transformations consisting of 
minimizing, restoring and maximizing the size of a window. 

11. A method as claimed in any of claims 8 to 10, wherein the typical 
transformations comprise changing a Z-order of the windows, according to 
which two or more of the windows are overlaid one upon another on the 
screen. 

12. A method as claimed in any of claims 8 to 10, wherein the typical 
transformations comprise changing a Z-order of the windows, according to 
which two or more of the windows are overlaid one upon another on the 
screen. 

13 . A method as claimed in any preceding claim, wherein encoding the 
description of the first set of windows comprises encoding the first -frame 
window characteristics and the respective contents of the windows in the 
first set, and wherein encoding the description of the second set of 
windows comprises encoding the determined transformations and encoding 



changes in the contents of the windows in the second set with respect to 
the contents of the corresponding windows in the first set. 

14. A method as claimed in claim 13, wherein encoding the changes in the 
contents of the windows comprises applying different encoding schemes to 
the contents of different ones of the windows. 

15. A method as claimed in claim 14, wherein applying the different 
encoding schemes comprises applying a video compression scheme to the 
contents of at least one of the windows, and embedding resultant compressed 
video data in the encoded description of the windows. 

16. A method as claimed in claim 14 or claim 15, wherein applying the 
different encoding schemes comprises applying different levels of encoding 
resolution to different ones of the windows in the second set. 

17. A method as claimed in any preceding claim, wherein identifying the 
first and second sets of windows comprises identifying windows associated 
with respective functions of the computer, wherein the identified windows 
include one or more user interface windows generated inside other 
identified windows for the purpose of controlling the functions associated 
therewith . 

18. A method as claimed in any preceding claim, and comprising 
identifying first and second sets of one or more icons in the first and 
second frames, respectively, and determining transformations applied to the 
icons in the first frame to generate the icons in the second frame, to be 
encoded along with the description of the second set of windows . 

19. A method as claimed in claim 18, wherein the first and second sets of 
icons comprise a cursor. 

20. A method as claimed in any preceding claim, wherein encoding the 
first and second descriptions comprises transferring the encoded 
descriptions over a communication link to a recipient computer. 

21. A method as claimed in claim 20, wherein encoding the descriptions 
comprises encoding the descriptions in a platform- independent format. 



22. A method as claimed in any preceding claim, wherein encoding the 
first and second descriptions comprises storing the encoded descriptions in 
a memory. 

23. A method as claimed in any preceding claim, and comprising 
reconstmcting the second frame responsive to the encoded descriptions of 
the first and second sets of windows. 

24. A method for reconstructing captured computer screens, comprising: 

receiving an encoded description of a first set of one or more 
windows, having first-frame characteristics and window contents, which 
appeared on the computer screen in a first captured frames- 
receiving an encoded description of a second set of one or more 
windows, having second-frame characteristics and window contents, which 
appeared on the computer screen in a second captured frame, subsequent to 
the first frame, the description of the second set of windows comprising a 
description of one or more transformations applied to the first-frame 
characteristics of at least one of the windows in the first set to derive 
the second- frame characteristics of a corresponding window in the second 
set ; and 

reconstructing the second captured frame responsive to the encoded 
descriptions of the first and second sets of windows. 

25. A method as claimed in claim 24, wherein reconstructing the second 
captured frame comprises decoding the encoded description of the first set 
of windows to determine the first-frame characteristics thereof, and 
applying the one or more transformations described in the description of 
the second set of windows to transform the first- frame characteristics into 
the second- frame characteristics of the at least one corresponding window. 

26. A method as claimed in claim 25, wherein the encoded description of 
the second set of windows further comprises encoded changes in the contents 
of the windows in the second set with respect to the contents of the at 
least one corresponding window in the first set, and wherein reconstructing 
the second captured frame comprises reconstructing the contents of the 
windows in the second set responsive to the encoded changes. 



27. A method as claimed in claim 25, wherein the encoded description of 
the second set of windows comprises compressed video data in a standard 
media format, and wherein reconstructing the second captured frame 
comprises invoking a standard media player to reconstruct video images in 
one of the windows , 

28. A method as claimed in claim 24, wherein reconstructing the second 
captured frame comprises reconstructing the first and second sets of 
windows substantially independently of an operating system under which the 
windows were generated. 

29. A method as claimed in claim 28, wherein reconstructing the first and 
second sets of windows comprises operating a platform- independent screen 
player , 

30. Apparatus for capture of computer screens in a sequence of frames, 
comprising : 

a display; and 

a processor, which is adapted to identify a first set of one or more 
windows appearing on the display in a first frame in the sequence, each 
window in the set having respective first-frame window characteristics and 
window contents, and to encode a description of the first set of windows, 
indicative of the appearance of the computer screen in the first frame, and 

to identify, in a second frame in the sequence a second set of one or 
more windows appearing on the display, having respective second-frame 
window characteristics and window contents, the second set including one or 
more windows corresponding respectively to one or more of the windows in 
the first set, and to determine one or more transformations applied to the 
first- frame window characteristics of the windows in the first set to 
generate the second -frame window characteristics of the corresponding 
windows in the second set, and 

to encode a description of the second set of windows including the 
determined transformations, for use in reconstructing the computer screen 
as it appeared in the second frame. 

31. Apparatus as claimed in claim 30, wherein the windows are generated 
in accordance with an operating system of the processor, which associates 



each window with a respective function of the processor, such that the 
contents of the windows are determined by the respective functions. 

32. Apparatus as claimed in claim 30 or claim 31, wherein the processor 
is further adapted to identify first and second sets of one or more icons 
in the first and second frames, respectively, and to determine 
transformations applied to the icons in the first frame to generate the 
icons in thei second frame, to be encoded along with the description of the 
second set of windows. 

33. Apparatus as claimed in any of claims 30 to 32, wherein the processor 
is adapted to be coupled via a communication link to transfer the encoded 
descriptions to a recipient computer. 

34. Apparatus as claimed in claim 33, wherein the recipient computer 
reconstructs the second frame responsive to the encoded descriptions of the 
first and second sets of windows. 

35. Apparatus as claimed in any of claims 30 to 34, and comprising a 
memory adapted to store the encoded first and second descriptions. 

36. Apparatus for reconstructing captured computer screens, comprising: 

a processor, which is adapted to receive an encoded description of a 
first set of one or more windows, having first-frame characteristics and 
window contents, which appeared on the computer screen in a first captured 
frame , and 

to receive an encoded description of a second set of one or more 
windows, having second-frame characteristics and window contents, which 
appeared on the computer screen in a second captured frame, subsequent to 
the first frame, the description of the second set of windows comprising a 
description of one or more transformations applied to the first- frame 
characteristics of at least one of the windows in the first set to derive 
the second- frame characteristics of a corresponding window in the second 
set, and 

to reconstruct the first and second captured frames responsive to the 
encoded descriptions of the first and second sets of windows; and 
a display, which is adapted to be driven by the processor to display the 
reconstructed first and second frames. 
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37. Apparatus as claimed in claim 36, wherein the processor is adapted to 
reconstruct the second captured frame substantially independently of an 
operating system under which the windows in the first and second frames 
were generated. 

38. Apparatus as claimed in claim 36, wherein the processor is adapted to 
reconstruct the second captured frame substantially independently of an 
operating system under which the windows in the first and second frames 
were generated. 

39. A computer program comprising computer program instructions to, when 
loaded into a computer system and executed, cause the computer to perform 
the steps af a method as claimed in any of claims 1 to 23 . 

40. A computer program comprising computer program instructions to, when 
loaded into a computer system and executed, cause the computer to perform 
the steps af a method as claimed in any of claims 1 to 23. 

41. A computer program as claimed in claim 38, wherein the program 
instructions are provided in a platform- independent format. 
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